Using the SAS System to Demonstrate the equivalence of On-line and On-paper Survey Administration across Levels of Raters

This research used logistic regression to model item responses from a popular 360-for-development survey. The model used method of survey delivery and rater group to identify items that exhibited differential item functioning (DIF). The methods of survey delivery were pencil-and-paper and online by web page. The rater groups were self, boss, peer, and direct report. The sample consisted of 374 survey families where a survey family consisted of a matched set of four surveys: one self, one boss, one peer, and one direct report. Half of the survey families were from a pencil-and-paper administration; half were from an online administration. The sample contained 1496 total surveys. The procedure to flag items exhibiting differential functioning used effect size computed from Wald chi-square statistics rather than statistical significance, resulting in fewer flagged items. The results indicated little evidence to suggest that rating differences exist due to the method of survey delivery. However, approximately 10% of the survey items exhibited differential item functioning attributable to rater group, though the effect size for each item was small. The examination of the maximum likelihood parameter estimates suggested that some of this differential functioning could be the result of hierarchical complexity. The anomalous functioning of other items was attributable to contingency theory. This research further suggested that such forms of DIF might be naturally occurring phenomena in some 360-assessment, and that the interpretation of 360-feedback may need to include the potential for this DIF to exist.